Support multiple groups and pipelines #1823

lquerel · 2026-01-19T06:20:34Z

This PR adds support for multiple pipeline groups and multiple pipelines per group. It is still possible to start the engine with a single pipeline, but it is now also possible to start the engine with a configuration file describing one or more pipeline groups. An example configuration is included in this PR.

The controller documentation has been updated to provide more details. Optimal placement of the ITS and pipelines per NUMA node is not yet implemented. Named channels are not implemented yet.

I did not observe any performance regression with this PR.

codecov · 2026-01-19T06:30:12Z

Codecov Report

❌ Patch coverage is 33.23944% with 237 lines in your changes missing coverage. Please review.
✅ Project coverage is 84.66%. Comparing base (107ef6b) to head (29ed40e).
⚠️ Report is 1 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1823      +/-   ##
==========================================
- Coverage   84.73%   84.66%   -0.08%     
==========================================
  Files         499      499              
  Lines      146658   146784     +126     
==========================================
- Hits       124276   124269       -7     
- Misses      21848    21981     +133     
  Partials      534      534

Components	Coverage Δ
otap-dataflow	`86.06% <33.23%> (-0.13%)`	⬇️
query_abstraction	`80.61% <ø> (ø)`
query_engine	`90.52% <ø> (ø)`
syslog_cef_receivers	`∅ <ø> (∅)`
otel-arrow-go	`53.50% <ø> (ø)`
quiver	`90.66% <ø> (ø)`

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

lalitb · 2026-01-20T07:29:29Z

Thanks for the PR. Minor question on core_count with multiple pipelines: I noticed the sample config uses core_set with explicit non-overlapping ranges for each pipeline.

Is core_set the recommended approach when running multiple pipelines that need core isolation?
When using core_count across multiple pipelines, is it expected that they may end up sharing the same cores?

albertlockett · 2026-01-20T14:49:47Z

rust/otap-dataflow/crates/config/src/engine.rs

+    /// Supports:
+    /// - JSON files: `.json`
+    /// - YAML files: `.yaml`, `.yml`
+    pub fn from_file<P: AsRef<Path>>(path: P) -> Result<Self, Error> {


In the future, maybe we'd want to consider the config crate for this? https://crates.io/crates/config . It might reduce some of the boilerplate loading config

There is indeed this repetition issue, but at the same time there is also this error reporting problem described here: #1832. I need to test this crate config and other approaches to see what will help resolve a number of issues around configuration parsing.

Sounds good. One thing that's nice about this config crate is that it's possible to override the config file with environment variables:
https://docs.rs/config/0.15.19/config/#example

This can be helpful for users who have some dynamic config, but don't want to set up the machinery to deploy a templated config file.

This sounds good. Whatever we do, let's not invent a crate for configuration 😂

I agree. I will check this crate and validate the quality of the error messages

lalitb

LGTM. Thanks for the PR. Had a nit question, not directly related to this PR changes.

lquerel · 2026-01-20T18:49:17Z

@lalitb Thanks for the PR. Minor question on core_count with multiple pipelines: I noticed the sample config uses core_set with explicit non-overlapping ranges for each pipeline.
1. Is `core_set` the recommended approach when running multiple pipelines that need core isolation?

2. When using `core_count` across multiple pipelines, is it expected that they may end up sharing the same cores?

This part of the system still needs improvement. I would like to reach a situation where:

Either the number of cores (without specific allocation) to use is defined in the pipeline configuration and the controller decides on the optimal placement.

Or the exact core allocation is defined in the pipeline configuration and the controller follows the configuration strictly.

I created the following GH issue for tracking: #1837

jmacd

Question: It's not obvious from the PR how the distinction between single-pipeline configuration and pipeline-group configuration is done. I see the EngineConfig type already existed, so I suspect that's making it hard to see how it all connects.

jmacd · 2026-01-20T18:54:26Z

rust/otap-dataflow/crates/config/src/engine.rs

+    /// Supports:
+    /// - JSON files: `.json`
+    /// - YAML files: `.yaml`, `.yml`
+    pub fn from_file<P: AsRef<Path>>(path: P) -> Result<Self, Error> {


This sounds good. Whatever we do, let's not invent a crate for configuration 😂

# Conflicts: # rust/otap-dataflow/crates/controller/src/lib.rs # rust/otap-dataflow/crates/state/src/store.rs

lquerel · 2026-01-20T19:16:05Z

@jmacd Question: It's not obvious from the PR how the distinction between single-pipeline configuration and pipeline-group configuration is done. I see the EngineConfig type already existed, so I suspect that's making it hard to see how it all connects.

For now, I have only added an extra parameter to the command line (--config, -c) to allow specifying the global engine configuration, and I have kept the parameter (--pipeline, -p) for starting a single pipeline.

lquerel added 3 commits January 16, 2026 07:28

Add engine config file with support for 1 pipeline group and 1 pipeline

25486ed

Add engine config file with support for 1 pipeline group and 1 pipeline

422bf78

Support multiple groups and nested pipelines

f19b994

github-project-automation bot added this to OTel-Arrow Jan 19, 2026

github-actions bot added the rust Pull requests that update Rust code label Jan 19, 2026

lquerel changed the title ~~Support multiple groups and pipelines~~ [WIP] Support multiple groups and pipelines Jan 19, 2026

lquerel self-assigned this Jan 19, 2026

lquerel added the engine-capability Internal engine features label Jan 19, 2026

Merge branch 'main' into controller-config

745cce0

lquerel added 7 commits January 18, 2026 22:45

Fix merge

5b931d0

Document engine config

6772950

A bit of refactoring in the main and the controller to make things nicer

e97c423

Improve the controller README.md

c7beeb0

Add architecture diagram and update main README.md

45f8d04

Update diagrams and documentation

e0e643c

A little bit of refactoring in the controller

14ea19e

lquerel changed the title ~~[WIP] Support multiple groups and pipelines~~ Support multiple groups and pipelines Jan 20, 2026

lquerel marked this pull request as ready for review January 20, 2026 00:20

lquerel requested a review from a team as a code owner January 20, 2026 00:20

albertlockett reviewed Jan 20, 2026

View reviewed changes

albertlockett approved these changes Jan 20, 2026

View reviewed changes

lalitb approved these changes Jan 20, 2026

View reviewed changes

Update configuration files

1ec9d6c

jmacd approved these changes Jan 20, 2026

View reviewed changes

lquerel added 2 commits January 20, 2026 11:06

Merge remote-tracking branch 'upstream/main' into controller-config

37c81d7

# Conflicts: # rust/otap-dataflow/crates/controller/src/lib.rs # rust/otap-dataflow/crates/state/src/store.rs

Merge with main upstream

e89820a

Fix markdown lint issues

29ed40e

lquerel enabled auto-merge January 20, 2026 19:23

lquerel added this pull request to the merge queue Jan 20, 2026

Merged via the queue into open-telemetry:main with commit 448d7ac Jan 20, 2026
43 of 44 checks passed

lquerel deleted the controller-config branch January 20, 2026 19:51

github-project-automation bot moved this to Done in OTel-Arrow Jan 20, 2026

Support multiple groups and pipelines #1823

Support multiple groups and pipelines #1823

Uh oh!

Conversation

lquerel commented Jan 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov bot commented Jan 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

lalitb commented Jan 20, 2026

Uh oh!

albertlockett Jan 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

lquerel Jan 20, 2026

Choose a reason for hiding this comment

Uh oh!

albertlockett Jan 20, 2026

Choose a reason for hiding this comment

Uh oh!

jmacd Jan 20, 2026

Choose a reason for hiding this comment

Uh oh!

lquerel Jan 20, 2026

Choose a reason for hiding this comment

Uh oh!

lalitb left a comment

Choose a reason for hiding this comment

Uh oh!

lquerel commented Jan 20, 2026

Uh oh!

jmacd left a comment

Choose a reason for hiding this comment

Uh oh!

jmacd Jan 20, 2026

Choose a reason for hiding this comment

Uh oh!

lquerel commented Jan 20, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

lquerel commented Jan 19, 2026 •

edited

Loading

codecov bot commented Jan 19, 2026 •

edited

Loading

albertlockett Jan 20, 2026 •

edited

Loading